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FOREWORD 


This  evaluation  plan  was  written  under  the  Career  Systems  Design  (CSYD)  project  of  the 
Education  and  Training  Function  Program  Element  0603720N.  The  work  is  sponsored  by  the 
Deputy  Chief  of  Naval  Operations  (OP-1 1 1 J). 

The  CSYD  project,  which  was  conceived  using  a  systems  approach  to  career  implementation, 
preparation,  and  development,  was  tasked  to  design  a  rating  training  continuum  design 
methodology.  The  resultant  methodology  was  applied  to  the  Operations  Specialist  (OS)  and  the 
Electronics  Warfare  Technician  (EW)  ratings.  Continuum  training  plans  were  completed  for  both 
ratings. 

The  plan  presented  here  is  intended  to  be  used  to  evaluate  the  effectiveness  of  the  OS  and  EW 
continuum  training  plans.  The  plan  addresses  four  attributes:  Job  performance,  force  structure, 
training  effectiveness,  and  career  attitude. 


J.  C.  McLACHLAN 
Director,  Training  Systems  Department 
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SUMMARY 


Background  and  Problem 

The  Deputy  Chief  of  Naval  Operations  (Manpower,  Personnel  and  Training  (OP-01)  tasked 
Navy  Personnel  Research  and  Development  Center  (NPRDC)  to  construct  a  rating  training 
continuum  design  methodology  using  the  Operations  Specialist  (OS)  and  Electronics  Warfare 
Technician  (EW)  ratings  as  the  design  vehicles.  Rating  Training  Continuum  Development 
Workshops  were  held  for  the  OS  and  EW  ratings  in  FY89  and  FY90.  The  results  of  the  workshop 
were  incorporated  into  Continuum  Training  Plans  (CTPs)  and  published  in  FY91. 

The  Continuum  Development  Office  (OP-1 11 J)  needed  a  plan  to  evaluate  the  effects  of  the 
CTPs  on  the  training  pipelines  of  the  OS  and  EW  ratings. 

Objective 

The  objective  of  this  effort  was  to  design  a  plan  to  evaluate  the  effectiveness  of  the  OS  and  EW 
CTPs. 

Evaluation  Method 

The  first  step  in  the  evaluation  of  the  CTPs  is  to  verify  that  the  recommendations  in  the  CTPs 
have  been  implemented  into  the  OS  and  EW  training  pipelines.  Implementation  will  be  established 
by  surveying  the  implementing  and  using  agencies,  and  examining  course  materials. 

Once  implementation  has  been  established,  it  will  be  possible  to  measure  the  effect  that  the 
recommendations  have  had  on  OS  and  EW  personnel,  and  on  the  training  pipelines.  Four  attributes 
will  be  measured:  Job  performance,  force  structure,  training  effectiveness,  and  career  attitude. 
Existing  organizational  data,  surveys,  and  interviews  will  be  used  to  obtain  the  necessary  data  on 
these  four  attributes.  Descriptive  and  inferential  statistics  will  be  used  to  analyze  the  data. 

Recommended  Evaluation  Plan 

The  evaluation  plan  identifies  the  variables  and  the  procedures  necessary  to  evaluate  the 
effectiveness  of  the  OS  and  EW  CTPs.  If  the  recommendations  in  the  CTPs  have  not  been 
implemented,  the  evaluation  must  be  discontinued. 

Baseline  data  should  be  collected  immediately.  Evaluation  data  should  '  e  collected  after 
completion  of  two  or  more  evolutions  of  the  courses  revised  as  a  result  of  the  CTPs.  Evaluation 
data  gathered  too  soon  will  not  reflect  the  changes  in  the  training  pipeline. 

Data  should  be  gathered  at  regular  intervals  following  the  initial  evaluation  to  determine  the 
continuing  effect  of  the  recommendations  across  a  30-year  career. 


The  tasks  that  need  to  be  completed  are  listed  below: 

1.  First  Year 

a.  Identify,  collect,  and  analyze  existing  baseline  data  immediately. 

b.  Develop  and  administer  surveys  to  obtain  data  not  currently  available. 

c.  Process  and  analyze  survey  data. 

d.  Combine  all  baseline  data  for  comparison  purposes. 

e.  Submit  report  on  baseline  data  to  OP- 1 1 1 J. 

2.  Second  Year 

a.  Verify  implementation  of  OS  and  EW  CTPs  using  implementation  variables. 

b.  Collect  and  analyze  existing  organizational  data  on  evaluation  variables. 

c.  Administer  surveys  to  obtain  data  not  currently  available. 

d.  Process  and  analyze  survey  data. 

e.  Compare  evaluation  data  to  baseline  data. 

f.  Report  results  to  OP- 1 1 1  J. 
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INTRODUCTION 


Background  and  Problem 

In  1983,  a  Navy  Inspector  General  (NAVINSGEN)  identified  several  training  areas  that  needed 
improvement.  Specifically,  improvement  was  needed  in  the  communications  between  the  fleet  and 
school  commands,  in  the  training  community's  response  time  to  fleet  comments,  and  in  the 
agreement  between  the  fleet-required  skills  and  knowledge,  and  the  school  curriculum.  The  rating 
training  continuum  concept  evolved  when  senior  Navy  decision  makers  concluded  that  no  cohesive 
and  comprehensive  plan  for  training  requirements  existed  across  the  career  of  enlisted  personnel. 
The  Navy  Training  Strategy  (Office  of  Chief  of  Naval  Operations,  1989)  directed  the  Deputy  Chief 
of  Naval  Operations  (Manpower,  Personnel  and  Training  (OP-01))  to  coordinate  the  application  of 
a  continuum  approach  to  rate  and  mission  area  training.  The  training  continuum  concept 
establishes  career  training  requirements  and  provides  an  appropriate  and  timely  mix  of  formal 
school,  on-board  training,  and  sea  and  shore  assignments. 

The  Navy  Personnel  Research  and  Development  Center  (NPRDC)  was  tasked  to  construct  a 
training  continuum  design  methodology  using  the  Operations  Specialist  (OS)  and  Electronics 
Warfare  Technician  (EW)  ratings  as  the  design  vehicles.  The  continuum  training  concept  requires 
the  participation  of  current  fleet  experienced  rating  qualified  subject  matter  experts  (SMEs)  to 
develop  the  continuum  training  plan  (CTP).  A  workshop  was  determined  to  be  the  most  effective 
means  of  bringing  the  SMEs  together  and  obtaining  the  necessary  information. 

To  date,  there  have  been  two  rating  training  continuum  development  workshops.  The  first  was 
for  the  OS  rating;  and  the  second,  for  the  EW  rating.  During  each  workshop,  the  SMEs  developed 
training  requirements  in  the  form  of  personnel  performance  profiles  (PPPs),  training  level 
assignments  (TLAs),  and  a  complete  training  pipeline.  All  the  information  was  then  incorporated 
into  a  separate  CTP  for  each  rating. 

The  Career  System  Design  (CS  YD)  rating  training  continuum  effort  is  nearing  completion  with 
only  two  remaining  development  phases.  They  are  the  implementation  and  evaluation  phases.  To 
assist  the  Continuum  Development  Office  (OP-1 11 J)  in  conducting  the  evaluation  phase,  an 
evaluation  planning  document  needed  to  be  developed. 

Objective 

The  objective  of  this  effort  was  to  design  a  plan  to  evaluate  the  effectiveness  of  the  OS  and  EW 
CTPs. 


EVALUATION  METHOD 

The  objective  of  an  evaluation  plan  is  to  define  and  develop  a  realistic  program  to  determine 
the  effectiveness  of  a  particular  program  or  intervention.  In  this  case,  the  intervention  consists  of 
the  recommendations  in  the  OS  and  EW  CTPs.  Before  an  evaluation  can  take  place,  it  is  necessary 
to  verify  that  the  recommendations  made  in  the  CTPs  have  been  implemented.  Hence,  the 
evaluation  must  first  verify  that  the  recommendations  in  the  CTPs  have  been  implemented  and  then 
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measure  the  effect  that  the  recommendations  have  had  on  OS  and  EW  personnel,  and  on  the 
training  pipelines. 

Design 

To  evaluate  the  effectiveness  of  the  OS  and  EW  CTPs,  a  plan  to  obtain  data  on  the  attributes 
must  be  established.  An  “experimental  design”  specifies  a  plan  for  assigning  subjects  to 
experimental  conditions  and  the  statistical  analysis  to  be  used  in  measuring  the  results.  True 
experimental  designs  have  control  groups  and  full  experimental  control  of  both  the  subjects  and 
their  exposure  to  the  intervention.  In  this  evaluation,  all  of  the  subjects  will  be  exposed  to  the 
intervention,  the  recommendations  in  the  CTP,  and  there  will  be  no  control  group.  The  subjects  will 
not  be  in  a  controlled  laboratory  environment,  but  in  the  field.  In  addition,  the  subjects  will  be 
exposed  continually  to  the  changes.  Exposure  to  the  intervention  will  not  be  an  isolated  event 
Because  of  these  constraints,  a  quasi-experimental  design  is  best  suited  to  this  evaluation. 

The  most  appropriate  quasi-experimental  design  for  this  evaluation  is  the  time-series  design. 
In  the  time-series  design,  the  attributes  of  a  particular  subject  group  are  observed  or  measured  over 
time,  exposed  to  the  intervention,  and  then  the  same  attributes  are  measured  again  over  time  to 
determine  the  effects  of  the  intervention.  The  time-series  design  assumes  that  the  exposure  to  the 
intervention  is  ongoing  after  the  initial  exposure,  as  in  this  evaluation.  It  also  assumes  that  group 
members,  although  subject  to  change,  retain  the  same  characteristics.  The  time-series  design 
provides  the  level  of  detail  and  scope  required  to  gain  a  comprehensive  understanding  of  the 
attributes  (Campbell  &  Stanley,  1966). 

Because  of  real-world  compromises  imposed  by  time,  funding,  and  data  gathering  constraints, 
this  evaluation  will  concentrate  on  the  most  useful  variables.  The  evaluation  will  include  surveys 
dedicated  to  measuring  the  criterion  variables,  database  development  and  analysis,  and  analysis  of 
organizational  data.  These  three  techniques  will  provide  a  multifaceted  approach  to  the  evaluation. 

Implementation  Measurement 

Implementation  is  “a  critical  process  in  the  life-cycle  of  a  new  program”  (Gray,  1984).  “The 
concept  of  degree  of  implementation  is  critical  in  order  to  derive  valid  conclusions  from  both 
outcome  and  process  studies  of  social  and  technological  innovations”  (Scheirer  &  Rezmovic, 
1983)  Without  empirical  evidence  of  the  degree  to  which  program  components  have  been 
implemented,  the  evaluator  of  an  innovative  program  will  have  difficulty  in  determining  the 
outcome  of  the  intervention. 

There  are  three  levels  of  implementation.  The  weakest  level,  adoptive  implementation,  refers 
to  rhe  process  surrounding  an  organization's  decision  to  try  out  the  innovation.  The  next  level, 
adaptive  implementation,  refers  to  changes  that  the  innovation  requires  to  improve  the  fit  with  the 
user  environment.  Finally,  the  most  robust  level,  programmed  implementation,  refers  to  changes 
that  the  user  environment  requires  for  the  innovation  to  be  used  routinely  (Scheirer  &  Rezmovic 
1983).  To  be  effective,  the  OS  and  EW  CTPs  must  achieve  programmed  implementation. 

Based  on  the  Training  Path  Charts  (TPCs)  developed  during  the  OS  and  EW  workshops,  and 
amended  by  fleet  review,  specific  recommendations  were  incorporated  into  the  continuum  training 
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plans  of  the  two  ratings.  To  determine  the  degree  of  implementation,  it  will  be  necessary  to 
ascertain  whether  the  recommendations  have  been  implemented.  This  will  involve  obtaining 
information  from  the  implementing  and  using  agencies  through  survey  and  organizational  data. 
The  information  will  provide  an  implementation  audit  trail,  as  well  as  determining  the  degree  of 
implementation. 

After  the  CTPs  have  been  formally  approved,  the  recommendations  will  be  implemented. 
However  baseline  data  should  be  collected  immediately  so  that  they  will  be  available  for 
comparison  with  the  data  collected  after  implementation. 

Table  1  lists  the  changes  recommended  to  the  OS  training  path  made  in  the  OS  CTP  and 
approved  by  fleet  and  schoolhouse  review. 

Table  2  lists  changes  recommended  to  the  EW  training  path  made  in  the  EW  CTP  and  approved 
by  the  fleet  and  schoolhouse. 


Table  1 

OS  Training  Path  System:  Recommended  Changes  (as  of  Oct  1990) 


Recommended 

Action 

Implementing 

Activity 

Using 

Activity 

Outcome 

Require  OS  intermediate 
and  advanced  courses 

PERS 

FCTCs 

New  courses 

Revised  pipeline 

Increase  ASAC  entry 
level  from  E-5  to  E-6 

PERS 

FCTCs 

Revised  pipeline 

Increase  ASAC  warfare 
tactical  training 

CNET 

PLATFORMS 

Revised  course  and 
on-board  training 

Use  OS  watchstanding  skills 
as  PCS  refresher 

PERS 

FCTCs 

Revised  pipeline 

Develop  5  new  PQS  items 

CNET 

PQSDEVGRU 

New  PQS  modules 

Reduce  length  of  AICS  course 

CNET 

FCTCs 

Revised  course 

Develop  MCM  navigation 
equipment  operator  course 

CNET 

FLEMINE 

WARTRACEN 

New  course 

Introduce  AEGIS 

NTDS  in  “A”  school 

CNET 

FCTCLANT  and 
AEGISTRACEN 

Revised  course 

Increase  IFF  training 

CNET 

FCTCs 

Revised  courses 

Note.  See  Glossary  for  definition  of  all  acronyms. 
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Table  2 


EW  Training  Path  System:  Recommended  Changes  (as  of  Jan  1991) 


Recommended 

Action 

Implementing 

Activity 

Using 

Activity 

Outcome 

Add  journeyman 
operations  training 

OP-392C 

CNET/CNTECHTRA , 

COMTRALANT, 

COMTRAPAC 

New  course 

Shift  ECCM  from  EWO  to 
CICO/FCO 

OP-762E 

Fleet 

Revised  pipeline 

Remove  NEC  1781  from  ships 
not  staff  capable 

PERS-22C6 

Fleet 

Revised  pipeline 

Change  NEC  1781  minimum 
prerequisite  from  E-6  to  E-7 

CNET/ 

Corry  Sta. 

CNET 

Revised  pipeline 

Untruncate  the  pipeline 

PERS-22C6 

Fleet 

Revised  pipeline 

Add  journeyman 
maintenance  course 

OP-392C 

CNET 

New  course 

Add  AEGIS  integration 
to  AN/SLQ-32  basic 
operations  course 

OP-392C 

CNET, 

COMTRALANT, 

COMTRAPAC 

Revised  course 

Add  Mk-36  Decoy  Launching 
Maintenance  training  to  4YO 

OP-392C/ 

PERS-22C6 

CNET 

Revised  course 

Move  6YO  advancement  to 
the  end  of  AN/SLQ-32  basic 
operations  course 

OP-392C/ 

OP-392C6 

CNET 

Revised  courses 

Note.  See  Glossary  for  definition  of  all  acronyms. 

Evaluation  Variables 

Once  implementation  has  been  established  by  surveying  the  implementing  and  using  agencies, 
and  examining  the  course  materials,  the  evaluation  can  begin.  An  evaluation  measures  certain 
attributes  that  reflect  the  changes  in  the  rating  before  and  after  implementation  and  determines  the 
effectiveness  of  the  rating  training  continuum  concept.  This  evaluation  will  measure  the  following 
four  attributes:  Job  performance,  force  structure,  training  effectiveness,  and  career  attitude 
(Training  Continuum,  1990).  Cost  attributes  are  not  included  in  the  evaluation  plan  because  they 
take  specialized  expertise  to  interpret  Each  attribute  consists  of  a  number  of  variables,  each  of 
which  measures  training  continuum  policy  as  well  as  diverse  evaluation  viewpoints.  Their  function 
is  to  provide  measurable  attributes  relating  continuum  policy  and  program  effectiveness. 

A  complete  list  of  the  possible  variables  and  the  criteria  used  to  select  them  appear  in 
Appendixes  A  and  B  respectively. 
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The  job  performance  variables  listed  in  Table  3  were  selected  to  determine  the  proficiency  of 
the  individual.  They  indicate  whether  or  not  the  revised  training  pipeline  is  producing  a  more 
effective  sailor. 


Table  3 

Job  Performance  Variables 


Variable 

How  Measured 

Instrument 

Supervisor  confidence 

Extent  work  meetVexceeds 
supervisor  expectation 

Interview  and / 
or  survey 

Skill  level 

Performance  level  attained  by  paygrade 

Surveys 

Supervision 

Amount  of  supervision  required 

Surveys 

PQS  progress 

PQS  performance 

Surveys 

Time  on  task 

Percent  time  spent  performing  tasks 

Surveys 

Recommendation  for  advancement 

Supervisors  stated  recommendation 

Surveys 

Note.  See  Glossary  for  definition  of  acronyms. 


The  force  structure  variables  listed  in  Table  4  have  been  selected  because  of  their  importance 
to  the  specific  rating.  They  highlight  the  personnel  resource  climate  confronting  the  OS/EW  rating 
detailers  and  the  fleet  they  serve.  Various  management  options  such  as  massive  force  reduction 
will  also  affect  these  components. 


Table  4 

Force  Structure  Variables 


Variable 

How  Measured 

Instrument 

Personnel  acquisition 

Number  of  acquisitions 

Recniitment 

Assignment 

Number  of  billets  filled  by  command 

Manpower  database 

Distribution  of  personnel 

Number  of  personnel  by  paygrade 

Personnel  database 

Turbulence 

Number  of  personnel  assigned  to  I  CSs/TDYs 

Manpower  database 

Attrition 

Number  of  personnel  leaving  Navy  before  EAOS 

Manpower  database 

Retention 

Number  of  reenlistments 

Manpower  database 

Advancement 

Number  of  personnel  eligible  for  advancement 

Manpower  database 

Note.  See  Glossary  for  definition  of  all  acronyms. 
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The  training  effectiveness  variables  listed  in  Table  5  were  selected  to  address  questions  of 
change  in  school,  curriculum,  and  general  student  academic  performance.  They  were  selected  to 
determine  the  impact  that  continuum  recommendations  have  on  the  overall  training  environment. 
Numerous  databases  are  available  including  TRAINTRACK  (Nakada,  Milczewsky,  &  Wax,  1989) 
and  the  Enlisted  Master  Record  (EMR). 


Table  5 

Training  Effectiveness  Variables 


Variable 

How  Measured 

Instrument 

Curriculum  revision 

Verify  course  Changes 

New  curriculum 

Course  length 

Check  course  syllabus  changes  and  schedule 

TRAINTRACK 

Course  prerequisites 

Verify  required  experience  and  paygrade 

Course  syllabus 

Training  attrition 

Number  of  attrites 

TRAINTRACK 

Training  backlog 

Number  of  days  awaiting  school 

TRAINTRACK 

Setbacks 

Number  of  academic  setbacks 

TRAINTRACK 

Training  interruptions 

Number  of  interruptions 

TRAINTRACK 

NECs  awarded 

CDPs  and  number  of  NECs  and  graduates 

TRAINTRACK 

Course  graduates 

Number  of  graduates 

TRAINTRACK 

Average  paygrade 

Compute  average  paygrade 

TRAINTRACK 

Average  number  on-board 

Compute  average  on-board 

TRAINTRACK 

Note.  See  Glossary  for  definition  of  all  acronyms. 

The  career  attitude  variables  listed  in  Table  6  are  relatively  soft;  that  is. 

they  are  based  on 

feelings,  opinions,  and  intentions  rather  than  on  performance.  They  are  easily  affected  by  personal 
and  environmental  changes. 

Table  6 

Career  Attitude  Variables 

Variable 

How  Measured 

Instrument 

Job  satisfaction 

Statistical  analysis  of  survey  data 

Survey 

Commitment 

Statistical  analysis  of  survey  data 

Survey 
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Data  Collection 


Both  implementation  and  evaluation  measurement  require  numerical  and  categorical  data. 
Most  studies  assessing  the  degree  of  implementation  use  more  than  one  measurement  technique. 
Dominant  techniques  include:  interviews,  surveys,  and  existing  databases  (Babbie,  1973). 

Interviews 

There  are  several  issues  to  consider  when  conducting  interviews.  The  selection  of  a  suitable 
interviewer  for  the  task  is  extremely  important  in  establishing  and  maintaining  respondent  and 
agency  confidence.  The  interviewer  must  also  be  able  to  illicit  the  required  information  without 
biasing  the  respondent  or  interfering  with  ongoing  activities.  In  addition,  it  will  be  important  to  this 
evaluation  to  maintain  cooperation  between  all  concerned  personnel  because  repeated  contacts  will 
be  necessary.  This  evaluation  will  use  face-to-face,  group,  and  telephone  interviews. 

Face-to-face  and  group  interviews  permit  good  subject  introduction,  stimulation  of  new  ideas, 
and  convenient  interaction  and  have  a  high  response  rate  (from  80%  to  85%). 

Telephone  interviews  are  inexpensive  since  they  minimize  travel  and  “no-answers”  can  be 
called  back.  Respondents  may  also  be  more  candid  over  the  telephone.  In  addition,  there  is  better 
access  to  senior  staff  members.  However,  the  fact  that  respondents  can  tire  during  lengthy  calls  can 
introduce  inaccuracies. 

Surveys 

Mailed  surveys  are  normally  inexpensive  to  administer.  They  can  deal  with  a  large  range  of 
questions  but  often  have  a  relatively  low  response  rate  (from  50%  to  60%).  The  estimated  response 
rate  will  affect  how  many  surveys  will  be  sent  to  the  fleet.  Another  consideration  in  this  evaluation 
is  that  before  a  survey  can  be  sent  to  the  fleet,  a  Report  Control  Symbol  must  be  obtained.  To  apply 
for  a  Report  Control  Symbol,  the  survey  instrument  must  be  completed,  and  the  sample  population 
identified.  The  time  involved  in  completing  the  paperwork  and  receiving  approval  must  be 
included  in  the  time  allotted  for  developing  a  survey. 

Database  Review 

Several  comprehensive  databases  available  at  NPRDC  can  provide  data  for  pre-  and  post¬ 
continuum  implementation  comparisons.  Searching  available  databases  is  less  intrusive  to  the 
Navy  commands  and  a  more  cost  effective  and  efficient  way  of  obtaining  the  information. 

TRAINTRACK  (Nakada  et  al.,1989)  contains  the  longitudinal  record  of  training  for  each 
individual  (service  attrites,  officers,  and  enlisted  personnel  from  all  services)  dating  back  to  1979. 
These  data  are  collected  at  the  end  of  each  fiscal  year  from  the  Student  Master  File  (SMF),  Survival 
Tracking  File  (STF),  and  CNET  cost  files  for  individuals  who  attended  a  school  that  reports  to  the 
Navy  Integrated  Resources  and  Administration  System  (NITRAS). 

The  Catalog  of  Navy  Training  Courses  (CANTRAC)  contains  information  on  Navy  schools 
and  courses.  CANTRAC  provides  a  consolidated  and  computerized  catalog  presenting  courses  in 
a  standardized  format.  The  data  include  course  descriptions,  convening  schedules,  course 
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prerequisites,  and  locations.  In  some  cases,  CANTRAC  also  provides  the  skill  profiles  associated 
with  the  courses. 

The  EMR  contains  career  information  for  every  enlisted  individual  in  the  Navy.  The  EMR 
provides  personal  and  service  data,  education  and  school  history,  classification  codes,  duty 
preferences  as  well  as  rate  and  rating  information. 

The  Navy  Personnel  Survey  Database  contains  attitudinal  information  on  job  satisfaction, 
organizational  climate,  training  and  education  programs,  and  other  career  related  issues. 

Subjects 

For  the  variables  that  are  not  available  from  an  existing  database,  it  will  be  necessary  to  obtain 
the  information  through  interviews  and/or  surveys.  The  subjects  who  are  surveyed  or  interviewed 
must  be  the  key  operational  personnel  most  capable  of  answering  individual  performance  and 
rating-specific  questions. 

Tables  1  and  2  identify  the  implementing  and  using  agencies  that  will  be  affected  by  the  OS 
and  EW  CTPs.  To  determine  if  the  recommendations  in  the  OS  and  EW  CTPs  were  implemented, 
personnel  from  the  agencies  listed  in  Tables  1  and  2  will  be  surveyed  or  interviewed  regarding  the 
criterion  variables. 

To  determine  the  effectiveness  of  the  OS  and  EW  CTPs,  fleet  personnel  as  well  as  personnel 
associated  with  the  specific  agencies  identified  in  Tables  1  and  2  will  be  surveyed  or  interviewed. 
Survey  candidates  will  include  those  whose  training  pipelines  were  affected  by  the 
recommendations  in  the  CTPs.  To  obtain  information  on  their  job  proficiency,  the  supervisors  of 
those  affected  by  the  CTPs  will  also  be  interviewed/surveyed.  Both  TRAINTRACK  and  the  EMR 
databases  will  be  useful  in  identifying  individuals  who  completed  the  affected  courses,  before  and 
after  implementation  of  the  CTPs. 

To  survey  all  of  the  approximately  13,000  OSs  and  3,000  EWs,  and  their  supervisors  would 
require  a  considerable  investment  of  both  time  and  resources.  To  avoid  this,  a  random  sample  of 
individuals  that  represent  the  entire  population  will  be  selected.  In  this  case,  a  sample  consisting  of 
10  percent  of  the  rating  population  is  the  minimum  number  of  subjects  required  to  provide  a 
representative  sample  (Gay,  1976).  Because  surveys  often  have  a  low  response  rate,  the  sample 
size  will  be  20  percent,  to  ensure  that  the  sample  is  not  too  small  to  generalize  to  the  total 
population. 

In  order  to  make  comparisons  and  draw  conclusions  about  the  effectiveness  of  the  OS  and  EW 
CTPs,  the  sample  of  fleet  and  agency  personnel  will  be  interviewed  or  surveyed  before  the  CTPs 
are  implemented  and  after  implementation  has  been  verified. 

Data  Analysis 

Evaluating  the  OS  and  EW  CTPs  will  involve  analyzing  many  aspects  of  the  program.  The 
most  appropriate  methods  for  analyzing  the  evaluation  data  are  descriptive  and  inferential 
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statistics.  Both  methods  exhibit  useful  and  achievable  traits  applicable  to  the  analysis  of  the 
performance  criteria  for  the  OS  and  EW  continua  (Tables  3  through  6). 

Descriptive  statistics  organize  and  summarize  data  to  describe  the  nature  of  a  population. 
Descriptive  statistics  generally  consist  of  measures  of  frequency,  location,  and  variability 
(Campbell  &  Stanley,  1966;  Kirk,  1968).  Descriptive  data  are  usually  collected  through  surveys  or 
interviews.  In  this  evaluation,  the  most  useful  descriptive  statistics  will  be  means  and  standard 
deviations. 

Inferential  statistics  use  statistical  information  derived  from  a  sample  population  to  draw 
conclusions  about  the  nature  of  the  entire  population  from  which  the  sample  was  drawn.  The  most 
useful  tests  of  significance  will  be  t-tests,  and  analyses  of  variance. 

RECOMMENDED  EVALUATION  PLAN 

Table  7  lists  the  tasks  needed  to  evaluate  the  effectiveness  of  the  OS  and  EW  CTPs  and  a 
suggested  time  line. 


Table  7 

Evaluation  Sequence 


Fiscal  Year 

1QTR  2QTR 

3QTR  4QTR 

First  Year 

Identify  resources  for  baseline  data 

— 

Collect  existing  baseline  data 

— 

Analyze  existing  baseline  data 

-- 

Develop  survey  instruments 

- 

Administer  survey  instruments 

— 

Process  survey  responses 

-- 

Analyze  survey  data 

— 

Combine  all  baseline  data 

— 

Submit  report  on  baseline  data  to  OP- 1 1 1 J 

— 

Second  Year 

Verify  implementation  of  OS  and  EW 

CTPs  using  implementation  variables 

~ 

Collect  existing  data  on  evaluation  variables 

— 

Analyze  existing  data  on  evaluation  variables 

-- 

Administer  survey  instruments 

- 

Process  survey  responses 

-- 

Analyze  survey  data 

- 

Compare  evaluation  data  to  baseline  data 

- 

Report  results 

- 
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The  evaluation  plan  has  identified  the  variables  and  the  procedures  necessary  to  determine  if 
the  recommendations  in  the  OS  and  EW  CTPs  have  been  implemented.  It  has  also  identified  the 
variables  and  the  procedures  necessary  to  evaluate  the  effectiveness  of  the  OS  and  EW  CTPs.  If 
the  recommendations  in  the  CTPs  have  not  been  implemented,  the  evaluation  must  be 
discontinued. 

Baseline  data  should  be  collected  immediately.  Evaluation  data  should  be  collected  after 
completion  of  two  or  more  evolutions  of  the  courses  revised  as  a  result  of  the  CTPs.  If  evaluation 
data  are  gathered  too  soon  they  will  not  reflect  the  changes  in  the  training  pipeline.  Data  should  be 
gathered  at  regular  intervals  following  the  initial  evaluation  to  determine  the  continuing  effect  of 
the  recommendations  across  a  30-year  career. 
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GLOSSARY 


AEGIS 

Advanced  Navy  Surface-to-Air  Missile  System 

AEGISTRACEN 

AEGIS  Training  Center 

AICS 

Air  Intercept  Controller  Supervisor 

ASAC 

Anti-Submarine  Air  Controller 

CANTRAC 

Catalog  of  Navy  Training  Courses 

CDP 

Course  Data  Processing  Code 

CICO 

Combat  Information  Control  Officer 

CNET 

Chief  of  Naval  Education  and  Training 

CNTECHTRA 

Chief  of  Naval  Technical  Training 

COMTRALA  NT 

Commander,  Training  Command,  U.S.  Atlantic  Fleet 

COMTRAPAC 

Commander,  Training  Command,  U.S.  Pacific  Fleet 

CSYD 

Career  Systems  Design 

CTP 

Continuum  Training  Plan 

EAOS 

Expiration  of  Active  Obligated  Service 

ECCM 

Electronic  Counter-countermeasures 

EMR 

Enlisted  Master  Record 

EW 

Electronics  Warfare  Technician 

EWO 

Electronics  Warfare  Officer 

FCO 

Fire  Control  Officer 

FCTC 

Fleet  Combat  Training  Center 

FCTCLANT 

FCTC  Atlantic 

FCTCPAC 

FCTC  Pacific 

FLEMINE 

WARTRACEN 

Fleet  Mine  Warfare  Training  Center 

IFF 

Identification  Friend  or  Foe 

MCM 

Mine  Countermeasures 

NAVINSGEN 

Navy  Inspector  General 

NEC 

Navy  Enlisted  Classification  code 

NITRAS 

Navy  Integrated  Resources  and  Administration  System 

NPRDC 

Navy  Personnel  Research  and  Development  Center 

NTDS 

Navy  Tactical  Data  System 

OS 

Operations  Specialist 

PCS 

Permanent  Change  of  Station 

PERS 

Bureau  of  Naval  Personnel 

PPP 

Personnel  Performance  Profile 
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PQS 

Personnel  Qualification  Standards 

PQSDEVGRU 

PQS  Development  Group 

SME 

Subject  Matter  Expert 

SMF 

Student  Mater  File 

STF 

Survival  Tracking  File 

TDY 

Temporary  Duty  Assignment 

TLA 

Training  Level  Assignment 

TPC 

Training  Path  Chart 

TRAINTRACK 

Enlisted  Training  Tracking  File 

4YO 

Four- year  obligation 

6Y0 

Six-year  obligation 
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APPENDIX  A 

DEFINITION  OF  POSSIBLE  EVALUATION  VARIABLES 
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DEFINITION  OF  POSSIBLE  EVALUATION  VARIABLES 
Job  Performance 

Supervisor  confidence.  Extent  to  which  an  individual’s  work  performance  meets  a 
supervisor's  expectations  on  rating  specific  and  nonrating  specific  tasks. 

Skill  level.  Performance  level  attained  by  paygrade. 

Supervision.  Amount  of  supervision  required  by  an  individual  in  performing  assigned  tasks. 

PQS  progress.  Extent  to  which  an  individual  completes  PQS  requirements. 

Time  on  task.  Time  spent  on  various  categories  of  rating  and  nonrating  specific  tasks. 

Recommendation  for  advancement.  Supervisor's  stated  commitment  to  recommend  an 
individual  for  advancement  based  on  overall  record  of  performance. 

Tasks  assigned  in  rate.  A  listing  of  the  types  of  rate  specific  tasks  performed  by  paygrade. 

Advancement  examination.  Collective  performance  score  on  advancement  exams  for  each 
rate. 

Force  Structure 

Personnel  acquisition.  Number  of  recruits  acquired  for  the  purpose  of  enlistment  or 
induction. 

Assignment.  Number  of  current  sea-based  and  shore-based  billets. 

Distribution  of  personnel.  Number  of  personnel  in  each  paygrade. 

Turbulence.  Events  which  interfere  with  normal  planned  and  scheduled  operations, 
processes,  or  activities,  such  as  the  number  of  personnel  assigned  to  PCS/TDYs. 

Attrition.  Number  of  individuals  who  separated  from  the  Navy  prior  to  their  Expiration  of 
Active  Obligated  Service  (EAOS)  date. 

Retention.  Number  of  individuals  who  reenlisted  upon  completion  of  their  EAOS. 
Advancement.  Number  of  personnel  eligible  for  advancement  to  a  higher  paygrade. 
Training  Effectiveness 

Curriculum  revision.  Extent  of  changes  in  course  content. 

Course  length.  Extent  of  changes  in  course  length. 

!See  Glossary  for  definitions  of  acronyms. 
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Course  prerequisites.  Change  in  the  minimum  level  of  prior  training,  experience,  and/or 
paygrade  required  to  attend  a  specific  course. 

Training  attrition.  Dissenrollment  from/or  unsatisfactory  completion  of  a  course  due  to 
academic  deficiency  or  for  nonacademic  reasons. 

Training  backlog.  The  number  of  days  an  individual  was  in  a  hold  status  awaiting 
enrollment,  or  having  enrolled,  is  awaiting  the  convene  date  of  a  scheduled  course. 

Setbacks.  The  number  of  academic  and  nonacademic  setbacks  an  individual  incurred  in  a 
particular  course. 

Training  interruptions.  The  total  and  average  number  of  times  an  individual  was  placed  in 
a  hold  status  when  instruction  was  interrupted  and  the  individual  was  unable  to  attend  after 
the  course  convening  date. 

NECs  awarded.  Number  of  Navy  Enlisted  Classification  (NECs)  codes  which  reflect  special 
knowledge  and  skills  that  identify  personnel  and  requirements  following  graduation  from 
an  NEC  producing  course. 

Course  graduates.  Number  of  individuals  who  have  successfully  completed  a  course  of 
instruction. 

Average  paygrade.  The  average  paygrade  of  individuals  enrolled  in  a  particular  course. 

Average  number  on-board.  Number  of  planned  or  actual  students  multiplied  by  the  number 
of  course-instruction  days  divided  by  the  number  of  days  in  a  year. 

Academic  acceleration.  The  total  number  of  times  an  individual  was  reassigned  to  an  earlier 
class  in  the  same  program  of  instruction. 

Standardization  of  training.  Extent  to  which  courses  taught  at  multiple  sites  (e.g., 
FCTCPAC  and  FCTCLANT)  match  each  other. 

Career  Attitude 

Job  satisfaction.  An  individual's  perceptions  of  satisfaction  with  their  job  in  particular  and 
the  Navy  in  general. 

Commitment.  An  individual's  perceptions  of  commitment  to  the  Navy  and  expressed 
intentions  to  complete  their  enlistment. 
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VARIABLE  SELECTION 


To  determine  which  variables  would  provide  the  most  information  and  would  be  the  least 
intrusive  and  difficult  to  obtain,  each  of  the  variables  listed  in  Appendix  A  was  rated  on  the  scale 
in  Table  B-L Table  B-l  lists  the  factors,  definitions,  and  rating  scales  used  to  determine  the  most 
appropriate  variables. 


Table  B-l 

Factors  of  Measurement  Efficiency 


Factor 

Definition 

Scale 

Relevance 

The  degree  to  which  variable  addresses 
training  continuum  performance. 

0-5 

Specificity 

Likelihood  that  variable  can  be  measured 
reliably  and  that  the  data  obtained  will 
reflect  measurement  requirements. 

0-5 

Cost 

The  monetary  cost  of  data  attainment. 

5-03 

Attainment 

difficulty 

The  problems  and  complexity  of  obtaining 
the  data. 

i 

O 

Attainment 

The  intrusiveness  necessitated  by  the 
data  gathering,  primarily  into  the 
shipboard  environment. 

5-03 

“These  factors  use  reverse  scoring;  i.e.,  low  cost,  low  attainment  difficulty,  low  intrusiveness  result  in  a  higher  score. 


B-l 


Table  B-2  lists  the  ratings  that  each  of  the  variables  attained  using  the  scale  in  Table  B-1 


Table  B-2 

Variable  Efficiency  Factor  Values 


Variable 

Relevance  Specificity 

Cost 

Difficulty  Intrusiveness 

Sum 

Job  Performance 

Superv  isor  confidence 

5 

5 

4 

4 

5 

23 

Skill  level 

5 

5 

4 

4 

5 

'  23 

Supervision 

5 

5 

5 

4 

4 

23 

PQS  progress 

5 

4 

3 

3 

5 

20 

Time  on  task 

5 

5 

4 

4 

5 

23 

Recommendation  for  advancement  5 

4 

2 

3 

2 

16 

Tasks  assigned  in  rate 

5 

5 

2 

2 

2 

16 

Advancement  exam 

5 

4 

2 

3 

2 

16 

Force  Structure 

Personnel  Acquisition 

4 

4 

5 

5 

5 

23 

Assignment 

4 

5 

4 

5 

5 

23 

Distribution  of  personnel 

5 

4 

5 

5 

5 

24 

Turbulence 

5 

4 

5 

4 

5 

23 

Attrition 

4 

4 

5 

5 

5 

23 

Retention 

3 

3 

5 

5 

5 

21 

Advancement 

3 

3 

5 

5 

5 

21 

Training  Effectiveness 

Curriculum  revision 

4 

4 

5 

5 

5 

23 

Course  length 

4 

4 

5 

5 

5 

23 

Course  prerequisites 

4 

4 

5 

5 

5 

23 

Training  attrition 

4 

4 

5 

5 

5 

23 

Training  backlog 

5 

4 

5 

5 

5 

24 

Setbacks 

4 

4 

5 

5 

5 

23 

Training  interruptions 

4 

4 

5 

5 

5 

23 

NECs  awarded 

4 

4 

5 

5 

5 

23 

Course  graduates 

4 

4 

5 

5 

5 

23 

Average  paygrade 

4 

4 

5 

5 

5 

23 

Average  number  on-board 

4 

4 

5 

5 

5 

23 

Academic  acceleration 

3 

3 

5 

5 

5 

21 

Standardization  of  training 

l 

1 

5 

5 

5 

17 

Career  Attitudes 

Job  satisfaction 

5 

5 

4 

5 

4 

23 

Commitment 

5 

5 

4 

5 

4 

23 
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